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Introduction 

This  project  was  a  mechanism  for  presenting  an  Online  Seminar  on  Cognitive  Modeling  for  User 
Interface  Design.  Two  experts  in  this  field,  David  Kieras,  at  the  University  of  Michigan,  and 
Bonnie  John,  at  Carnegie-Mellon  University,  prepared  slides  which  seminar  attendees 
downloaded  in  advance  from  a  web  site,  and  which  were  then  presented  as  a  seminar  online  over 
three  days  by  telephone  conference  call  and  streaming  audio  over  the  web,  and  questions 
collected  during  and  after  the  presentations  by  telephone  and  real-time  email  and  chat,  which 
were  then  answered  by  both  experts.  A  large  number  of  people  attended  the  seminar  and 
provided  by  feedback.  The  complete  materials  for  the  Online  Seminar  are  now  archived  and 
accessible  on  the  web  at  http://www.cs.cmu.edu/~bei/CognitiveModelingForUIDesign/ 

The  basic  rationale  for  the  seminar  is  that  the  two  experts  had  both  been  sponsored  in  their  past 
and  current  work  by  ONR  which  had  resulted  in  amassing  considerable  expertise  about  this 
increasing  important  topic,  and  so  this  knowledge  should  be  disseminated  more  widely.  Thus,  the 
two  presenters  contributed  their  time  and  effort  at  no  cost.  The  funding  in  this  project  was  to 
cover  all  of  the  expenses  of  presentation,  but  most  importantly,  to  gain  access  to  the  consulting 
services  for  planning,  publicity,  presentation  revision,  coaching,  arranging  web  and  telephone 
resources,  and  on-line  moderation  and  management  during  the  course  itself  by  an  expert  in 
online  teaching  and  learning,  Dr.  Lisa  Neal. 

Thus  the  PI  and  the  University  of  Michigan  simply  administered  the  funding  for  this  project  (at  a 
greatly  reduced  indirect  cost  rate),  and  Dr.  Neal  did  the  substantive  work.  Dr.  Neal  brought  the 
project  in  well  under  budget  (the  remaining  funds  were  returned  to  ONR).  Dr.  Neal  also  prepared 
a  detailed  and  complete  final  report  for  the  PI,  containing  information  about  the  attendance  and 
the  lessons  learned.  Because  of  its  value  in  future  such  activities,  rather  than  excerpt  or 
paraphrase  Dr.  Neal’s  excellent  report,  it  is  simply  included  in  the  following  pages. 
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Final  Report 

Seminar  on  Cognitive  Modeling 
for  User  Interface  Design 


Lisa  Neal,  Ph.D. 

3  Valley  Road 
Lexington,  MA  02421 
Phone:  781-861-7373 
Fax:  781-862-7890 
Email:  lisa@acm.org 


1.  Project  overview 


The  project  was  to  deliver  a  free  online  seminar  taught  by  Professor  David  Kieras  and  Professor  Bonnie  John  on 
Cognitive  Modeling  for  User  Interface  Design.  We  decided  in  advance  the  best  structure  was  to  offer  it  for  3 
consecutive  days,  February  28,  March  1 ,  and  March  2,  2005,  for  2.5  hours  each  day.  We  decided  to  use  a  low-tech 
approach  of  audioconferencing  by  phone  and  slides  that  participants  follow  on  their  own  to  meet  the  needs  of  people 
at  military  sites  where  software  can  not  be  installed  by  individuals.  We  also  decided  that  it  would  consist  primarily  of 
lecture  although  we  wanted  some  interactivity  through  questions  from  participants.  We  capped  enrollment  at  200.  We 
decided  that  we  would  archive  the  seminar  to  allow  for  replay  by  anyone  who  missed  all  or  part.  Finally,  we  decided 
to  get  feedback  from  participants  through  a  survey. 

2.  Project  deliverables  and  analysis 

2. 1  Seminar  topic 

Clearly,  from  the  number  of  people  signing  up  and  attending  the  seminar,  there  was  great  interest  in  the  topic  and  in 
learning  about  this  topic  from  these  well-known  and  well-respected  presenters.  There  is  also  interest  in  having  more 
seminars  in  the  future.  With  conference  travel  harder  to  get  funded,  and  tutorials  so  expensive  at  many  conferences, 
there  is  an  educational  need  that  this  seminar  clearly  filled. 

2.2  Seminar  style 

For  this  type  and  amount  of  material,  a  seminar  that  was  primarily  lecture  was  effective,  especially  since  there  were 
opportunities  for  asking  questions  and  additional  resources  were  provided.  For  the  number  of  people  attending,  it  was 
highly  interactive  and  effective  and  both  presenters  were  clear  and  articulate.  Prof.  Kieras  and  I  had  discussed  how  to 
present  effectively  online  before  the  seminar.  While  it  wasn’t  planned  this  way,  having  Prof.  John  available  to  answer 
questions  in  the  chat  and  by  phone  at  the  end  of  a  unit  during  Prof.  Kieras’  sessions,  and  having  Prof.  Kieras  do  the 
same  during  Prof.  John’s  session,  worked  very  well  and  made  the  sessions  more  interactive,  more  interesting,  and 
even  gave  people  a  bit  of  variety  in  the  voices  they  were  listening  to. 

I  acted  as  moderator  during  the  sessions,  starting  and  ending  each  session,  introducing  speakers,  thanking  the 
sponsor,  and  moderating  the  Q&A  segments.  I  also  offered  support  to  participants,  and  found  that  the  volume  of 
email  was  quite  high,  and  this  was  time-consuming  since  I  tried  to  answer  every  question  (see  comment  in  section 
2.4  below  about  need  for  a  FAQ  to  reduce  questions).  The  only  time  this  proved  especially  burdensome  was  when 
people  were  calling  me  or  emailing  me  right  at  the  start  of  a  session,  and  I  couldn’t  multitask  sufficiently  to  meet 
everyone's  needs  instantly. 

2.3  Seminar  duration  and  schedule 

The  timing  of  each  day  worked  very  well,  with  a  2.5  hour  session  covering  2  topics  per  session,  each  topic  followed 
by  Q&A,  and  a  5  minute  break  in  between.  The  seminar  ended  on  time  each  day,  only  going  over  a  few  minutes  on 
the  last  day  to  try  to  get  all  questions  answered.  I  don’t  believe  another  structure  of  condensing  it  into  fewer  days  or 
using  shorter  durations  for  more  days  would  have  worked  as  well;  certainly  it  is  hard  to  imagine  offering  the  seminar 
in  one  day  with  more  breaks.  Since  attendance  fluctuated  some,  and  it  may  have  been  hard  for  people  to  commit 
time  over  three  days,  having  a  shorter  period  each  day  for  more  days  may  have  led  to  fewer  people  attending. 
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The  time  of  day,  starting  at  1  p.m.  EST,  worked  well  for  all  US  time  zones,  and  for  people  in  Europe,  There  were 
many  interested  people  from  Asia/Pacific  who  probably  used  the  archive  instead  of  connecting  live. 

2.4  Seminar  announcement ,  registration,  web  sites,  and  attendance 

With  Prof.  Kieras  and  Prof.  John’s  help,  I  prepared  a  seminar  announcement  (see  Appendix  A)  that  was  distributed  to 
military  people  first  and  then,  a  week  later,  to  the  CHI  announcements  listserv  and  the  CTA  Resource  and  Cognitive 
Science  Society  mailing  lists.  The  announcement  was  designed  to  provide  all  the  information  a  potential  participant 
needed  to  know  about  what  the  seminar  covered,  how  it  was  being  offered,  who  was  offering  it,  and  what  it  cost  in  a 
length  that  could  be  distributed  by  email. 

Because  I  wanted  to  have  a  site  up  when  announcements  went  out  so  that  people  could  register,  I  decided  to  do 
what  was  quickest,  using  an  existing  service  with  a  template  to  configure  a  site.  I  chose  this  instead  of  having  a 
designer  build  a  site  for  us,  as  specified  in  the  proposal,  because  that  would  have  delayed  getting  the 
announcements  out,  and  we  needed  to  give  people  time  to  arrange  their  work  schedules  to  fit  the  seminar  in. 

In  choosing  between  a  number  of  options,  I  decided  the  Yahoo  Groups  site  had  many  advantages,  including  that  it 
someone  had  to  register  to  receive  messages  and  access  materials;  that  it  allowed  me  to  send  out  messages  to 
everyone  who  was  registered;  that  it  had  a  repository  for  shared  documents;  and  that  it  had  a  chat  tool  that  would 
work  on  military  configurations.  I  had  also  used  it  for  previously  and  knew  how  to  configure  it  quickly  and  easily.  The 
site  is  located  at  http://aroups.vahoo.com/qroup/CoanitiveModeling/. 

Yahoo  Groups  had  some  limitations  that  I  discovered  during  the  registration  process  and  during  the  seminar.  One 
limitation  was  that  some  people  had  Yahoo  blocked  at  work,  so  that  they  had  to  access  it  from  a  home  computer  to 
register  for  the  seminar  or  access  materials.  Another  limitation  was  that  the  site  only  allowed  a  certain  (unspecified) 
number  of  people  to  download  materials  daily.  The  final  limitation  was  the  storage  size;  while  all  of  Prof.  Kieras’ 
presentation  materials  and  the  various  seminar  announcements  could  be  posted,  Prof.  John’s  slides  and  the  archive 
of  the  audio  from  each  day  were  too  big.  I  discovered  the  limitation  about  restricting  the  number  of  people  who  could 
download  once  Prof.  Kieras’  materials  were  posted  and  we  reached  that  limit.  One  of  the  people  who  had  registered 
for  the  seminar,  Dan  Zlotnikov  at  the  University  of  Waterloo,  offered  to  set  up  a  mirror  site  with  the  presentation 
materials.  When  it  turned  out  that  the  audio  archive  was  too  large  to  post  at  the  Yahoo  site,  Dan  also  posted  them  on 
his  site  at  the  end  of  each  day. 

Following  the  seminar,  Prof.  John  set  up  a  permanent  archive  site  with  my  assistance  (writing  the  text  and  organizing 
the  materials  by  session).  While  the  Yahoo  Groups  site  is  still  there,  the  site,  at  http://www- 
2.cs,cmu.edu/--bei/CoqnitiveModelinqForUIDesiqn/index.htm.  provides  all  the  presentation  materials  and  audio 
archive.  Unfortunately,  I  did  not  think  about  including  a  counter  on  the  archive  site  until  after  it  was  announced,  so  we 
do  not  know  how  many  people  have  used  it. 

In  terms  of  numbers,  292  people  registered  at  the  Yahoo  Groups  site  and  over  300  contacted  me  about  the  seminar, 
either  to  register  or  ask  for  further  information.  (One  lesson  learned  in  this  process  is  that  no  matter  how  clearly 
something  is  stated,  at  least  one  person  will  not  understand  and  will  ask  about  it.  I  spent  a  lot  of  time  answering 
emails  with  specific  questions  about  different  aspects  of  the  seminar.)  The  number  of  people  who  came  to  the 
seminar  peaked  at  225  on  the  first  day.  The  second  day  attendance  peaked  at  175,  and  120  on  the  third  day.  This 
drop-off  was  not  surprising  given  the  number  of  hours  needed  to  take  the  entire  seminar  and  also  the  availability  of 
the  archive.  Also,  note  that  some  of  these  “connections”  were  groups  of  people  attending  together  (see  question  1 8  in 
the  survey,  Appendix  B,  for  further  information).  Overall,  this  was  an  excellent  response  for  an  online  seminar,  and 
exceeded  my  expectations. 

While  the  choice  of  the  Yahoo  Groups  site  was  expedient  and  provided  the  text  chat  during  the  seminar,  setting  up  a 
site  from  the  start  without  the  limitations  of  Yahoo  Groups  would  have  worked  better,  especially  that  only  a  certain 
number  of  people  could  download  files  every  day  and  that  the  audio  archive  files  were  too  large  to  post  there.  It  is 
unfortunate  these  limitations  were  not  documented.  It  still  would  have  been  necessary  to  have  a  mirror  site  since  too 
many  people  trying  to  download  files  at  once  caused  problems  with  the  other  sites  as  well.  Ideally,  the  site  for  course 
registration  should  have  been  the  same  as  the  permanent  archive  site.  The  other  thing  I  would  do  differently  is  create 
a  page  of  frequently  asked  questions  (FAQ)  because,  at  some  point,  I  received  a  question  about  every  aspect  of  the 
seminar,  including  “How  much  does  it  cost?”,  "Can  I  register  for  only  one  day?”,  and  “Is  each  day  a  repeat  of  the 
previous  day?”  Overall,  the  Yahoo  Groups  site  worked  well  and  can  be  kept  up  indefinitely,  as  can  the  final  archive 
site. 

2.5  Seminar  delivery 

2.5.1  Audioconferencing 

The  original  plan  was  to  provide  audioconferencing  by  phone  and  a  site  for  downloading  presentation  materials. 
Because  the  number  of  people  who  would  register  (or  who  would  actually  attend)  was  unknown  and  because  we 
were  focused  on  the  military,  we  did  not  consider  options  like  streaming  audio  until  part  way  through  the  registration 
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process.  For  the  phone  services,  I  wanted  one  that  provided  high  quality  audio;  put  people  on  automatic  mute, 
necessary  to  avoid  background  noise;  allowed  at  least  3  people  (Prof.  Kieras,  Prof.  John,  and  me)  to  be  on  the  open 
line,  was  operator-assisted  so  we  could  have  Q&A,  and  produced  an  archive. 

I  researched  audioconferencing  services  and  they  varied  considerably  in  price.  Phone  services  become  more 
expensive  the  more  services  you  add,  and,  for  some  companies,  there  was  a  large  per  minute  increase  when  an 
operator  was  added  to  a  call.  The  phone  service  I  selected,  Link  Conference  Call,  was  recommended  to  me, 
providing  an  assurance  of  high  quality  audio,  and  offered  operator-assisted  toll-free  calls  at  .12  per  person  per 
minute.  Other  companies  provided  the  same  service  for  as  much  as  .36  per  person  per  minute.  Link  Conference  Call 
was  willing  to  charge  for  actual  use  instead  of  estimated  use.  Since  more  people  always  sign  up  for  free  seminars 
than  actually  attend,  especially  when  an  archive  is  available,  I  wanted  to  avoid  having  to  estimate  how  many  of  the 
registered  people  would  really  attend  and  either  pay  for  people  who  weren’t  there  or  have  people  unable  to  get  into 
the  call.  Also,  it  didn’t  seem  that  we  would  know  until  the  last  minute  how  much  interest  there  was  and  we  needed  an 
option  that  would  work  for  a  small  to  large  (initially  capped  at  200)  group. 

The  service  was  configured  exactly  as  we  anticipated  would  work  best  for  this  type  of  seminar:  a  person  calling  in 
was  placed  on  automatic  mute  and  we  had  an  operator  available  who  could  open  the  line  for  questions  when  we 
requested  it.  The  operator  we  had,  Marie,  used  AOL  instant  messenger  (AIM),  and  she  and  I  were  able  to  IM  during 
the  sessions,  which  proved  helpful  since  I  could  tell  her  when  we  were  about  to  open  the  line  for  questions  and  she 
could  tell  me  how  many  people  were  connected  by  phone  or  streaming  audio  and  how  many  people  were  in  the 
queue  during  Q&A  sessions.  But  more  important  than  the  charges  and  services,  was  that  the  audio  quality  was 
excellent  throughout  the  3  days. 

2.5.2  Streaming  audio 

For  people  outside  of  the  US  and  Canada,  we  had  a  non-toll-free  number  they  could  use  to  join  the  seminar. 
However,  when  a  large  number  of  people  outside  of  the  US  and  Canada  expressed  interest  in  the  seminar,  we 
considered  adding  a  streaming  audio  option  since  otherwise  the  phone  charges  would  be  high  for  people  dialing  in. 
We  realized  that  people  using  streaming  audio  would  have  the  disadvantage  of  not  being  able  to  ask  questions 
verbally,  however  they  would  have  other  options  for  asking  questions. 

The  streaming  audio  option,  given  to  participants  as  a  url,  proved  to  be  more  popular  than  the  phone  option.  For 
people  in  the  US  and  Canada,  this  was  in  part  because  far  more  people  have  a  headset  for  their  computer  than  for 
their  phone,  making  listening  easier  and  less  disruptive  to  people  around  them;  also,  it  didn’t  tie  up  a  phone  line  for  so 
long.  Once  we  saw  how  high  the  level  of  interest  was  in  the  seminar,  and  decided  that  perhaps  we  shouldn’t  cap  it  at 
200,  as  previously  planned,  we  realized  that  if  everyone  who  expressed  interest  attended,  we  would  go  over  budget. 
Adding  the  streaming  audio  option  was  financially  advantageous  for  us  since  streaming  audio  was  offered  at  a  flat 
rate.  When  I  inquired  about  the  streaming  audio  option,  it  was  originally  quoted  at  $450  per  session.  The  manager 
then  contacted  me  the  next  day  to  say  that  he  had  forgotten  it  was  a  2.5  hour  call  and  the  rate  would  actually  be 
$650.  Since  I  had  already  gotten  Dr.  Chipman’s  ok  on  adding  this  option  and  incurring  the  expense,  I  suggested  we 
compromise  on  $550  per  session,  which  the  manager  agreed  to. 

While  there  were  some  minor  problems  with  streaming  audio,  there  were  few  complaints.  Interestingly,  participants 
are  more  tolerant  of  problems  with  streaming  audio  than  with  the  phone,  probably  because  people  don’t  expect  the 
perfect  quality  we  are  accustomed  to  on  the  phone  (other  than  cell  phones).  There  were  3  problems  that  occurred 
with  the  streaming  audio:  on  the  first  day  there  were  some  points  when  the  audio  quality  was  poor;  on  the  second  day 
the  person  running  the  streaming  audio  for  us  didn’t  realize  we  were  taking  a  break  and  ended  the  session,  but 
realized  his  mistake  and  reopened  it.  This  required  people  to  reconnect  once  they  realized  the  session  had  started  up 
again  but  they  weren't  hearing  it;  and  on  the  last  day  the  url  they  sent  me  that  I  distributed  to  everyone  had  a 
where  it  needed  a  “=’’.  Link  Conference  was  apologetic  about  all  3  issues,  especially  the  second  one,  and  they 
reduced  our  fee  for  that  day.  The  only  other  point  about  the  streaming  audio  is  that  their  web  site  should  have 
specified  that  someone  needed  Windows  Media  Player  and  where  to  download  it  from  (especially  for  Mac  users,  who 
may  not  have  it  already). 

While  overall  the  combination  of  phone  and  streaming  audio  worked  well,  the  only  thing  I  would  have  done  differently 
is  figured  out  a  tactful  way  to  promote  the  streaming  audio  option  over  the  phone,  for  those  who  could  use  it,  to 
reduce  cost.  I  also  would  have  provided  information  about  headsets  for  streaming  audio  users. 

2.5.2  Q&A 

We  provided  a  few  ways  that  people  could  ask  questions.  The  first  one  was  over  the  phone  at  the  end  of  each  unit, 
and  for  that  we  needed  an  operator  assisted  call.  That  worked  well,  especially  since  the  person  asking  the  question 
could  seek  clarification  or  follow  up  immediately  after  the  answer.  It  had  the  added  benefit  of  varying  the  voices 
people  were  listening  to. 
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The  second  way  of  asking  questions  was  through  email  to  me.  That  also  worked  well,  and  meant  that  any 
unanswered  questions  could  easily  be  forwarded  to  Prof.  Kieras  to  answer  following  the  session  (this  only  happened 
one  day). 

The  third  way  was  through  text  chat,  which  was  included  in  the  Yahoo  Group  I  set  up  for  seminar  participants.  The 
chat  proved  to  be  very  successful  in  many  respects,  particularly  that  Prof.  John  was  in  the  chat  while  Prof.  Kieras  was 
speaking  (and  Prof.  Kieras  while  Prof.  John  was  speaking)  answering  questions  and  providing  references  throughout 
the  seminar,  not  just  during  the  Q&A  at  the  end  of  a  unit.  Some  participants  answered  questions  for  each  other  as 
well.  The  chat  also  made  the  seminar  seem  much  more  interactive,  and  provided  more  of  a  sense  of  awareness  of 
others  through  the  participant  list  on  the  side.  The  chat  tool  was  selected  because  it  did  not  require  any  downloads 
and  should  work  on  military  configurations,  however  it  had  2  undocumented  limitations  that  impacted  its  usefulness. 
The  first  was  that  it  had  a  limit  of  between  40  and  50  participants,  so  once  the  limit  was  reached  others  couldn’t  get 
in.  A  higher  limit  or  no  limit  would  have  been  much  better.  The  other  limitation  was  that  there  was  no  way  to  save  the 
text  in  the  chat  other  than  with  screen  shots.  Ideally,  I  could  have  forwarded  the  questions  and  discussion  to  the 
presenters  following  the  session  and  also  archived  it. 

The  Q&A  went  well  and  made  the  seminar  more  interactive.  Offering  multiple  ways  of  asking  questions  worked  well 
too.  The  only  thing  I  would  have  done  differently  was  providing  a  text  chat  that  did  not  restrict  the  number  of 
participants,  and  capturing  the  questions  (and  answers)  on  the  web  site. 

2.5.3  Course  materials 

I  made  suggestions  to  Prof.  Kieras  about  making  his  presentation  materials  work  better  for  an  online  seminar,  since 
they  were  originally  developed  for  a  face-to-face  one.  The  course  materials  from  Prof.  Kieras  were  available  in 
advance  on  the  Yahoo  Groups  site  as  pdfs  to  download  and  there  were  no  problems  reported.  Prof.  John’s  materials 
were  very  large  files,  and  they  caused  problems  because  of  the  size.  We  ended  up,  at  the  last  minute,  creating  other 
mirror  sites.  (As  Dr.  Chipman  commented,  redundancy  can  be  beneficial,  especially  when  so  many  people  are 
accessing  large  files  at  once  to  download.) 

The  slides  worked  well,  very  extremely  informative  and  easy  to  follow,  and  the  presenters  did  an  excellent  job  of 
reminding  people  which  slide  they  were  on.  The  only  problem  was  due  to  the  size  of  Prof.  John’s  slides  for 
downloading. 

2.6  Survey 

I  created  a  survey,  reviewed  by  all,  and  posted  it  using  SurveyMonkey,  an  online  survey  tool.  102  participants  filled 
out  the  survey,  which  is  a  good  response  rate.  Feedback  was  very  positive  and  is  displayed  in  Appendix  B. 

2.7  Final  recommendations 

Overall,  this  was  a  great  success,  with  lots  of  positive  feedback  and  minimal  complaints.  We  came  in  under  budget, 
despite  greater  than  anticipated  interest,  due  to  the  use  of  streaming  audio.  There  were  some  unplanned  aspects  that 
worked  wonderfully,  such  as  each  presenter  helping  out  in  the  chat  while  the  other  presented,  and  each  helping 
answer  questions  at  the  end  of  each  unit.  It  was  also  helpful  to  be  able  to  instant  message  (IM)  with  Prof.  John  during 
the  sessions. 

The  only  thing  that  remains  to  be  done  is  to  announce  the  availability  of  the  archive  site,  possibly  through  the  same 
ways  the  seminar  was  announced  originally,  and  consider  the  requests  for  additional  seminars,  using  the  lessons 
learned  above.  Thank  you  for  giving  me  the  opportunity  to  support  this  seminar.  It  was  a  pleasure  working  with  Prof. 
Kieras  and  Prof.  John  and  I  appreciate  all  the  hard  work  they  put  into  making  this  a  success. 


Appendix  A.  Seminar  announcement 


Seminar  on  Cognitive  Modeling  for  User  Interface  Design 

Seminar  description:  This  seminar  presents  the  current  state  of  the  art  of  evaluating  user  interface  designs  using 
models  of  human  performance  that  are  based  on  cognitive  architectures.  Such  models  can  yield  usability  results 
without  the  delay  and  expense  of  user  testing  of  prototypes,  but  because  they  are  new  and  still  under  development, 
whether  and  how  to  apply  them  is  a  challenge.  This  seminar  will  survey  current  theory  and  practice;  no  "how-to"  of 
actual  model  construction  will  be  presented;  rather  the  goal  is  to  enable  a  good  choice  of  whether  a  modeling 
approach  will  be  useful,  and  which  type  of  model  would  be  best  to  pursue. 

Topics  will  include: 

•  Approaches  to  model-based  evaluation  of  user  interfaces 

•  An  example  cognitive  architecture 

•  Survey  of  cognitive  architecture  systems 

•  GOMS  models  as  simplified  cognitive  architecture  models 

•  Practical  issues  in  human  performance  modeling 

•  Some  current  tools  for  rapid  model  construction 

Seminar  Objectives:  In  this  seminar,  you  will  become  familiar  with 

•  The  basic  approaches  to  model-based  evaluation  of  user  interfaces,  and  their  advantages  and  disadvantages. 

•  The  important  features  of  cognitive  architectures  that  can  be  applied  to  evaluation,  and  specific  properties  of 
important  current  architectures. 

•  What  type  of  model  is  likely  to  work  best  in  a  particular  application. 

•  How  to  deal  with  practical  issues  of  model  construction,  validation,  and  usage,  using  some  current  tools. 
Seminar  dates:  February  28,  March  1 ,  and  March  2,  2005  from  1  -  3:30  p.m.  EST 

How  this  seminar  will  be  offered:  This  online  seminar  will  be  delivered  over  the  phone.  Seminar  materials  will  be 
available  online.  Each  seminar  session  will  provide  opportunities  for  Q&A,  and  questions  can  be  asked  using  the 
phone  or  email.  An  archive  of  each  session  will  be  available  following  the  session. 

Fees:  None 

Target  audience:  This  seminar  will  be  of  interest  to  human-computer  interaction  researchers  or  designers  who  want 
an  current  overview  of  the  modeling  approach,  or  who  are  considering  applying  this  approach  to  future  projects.  Prior 
background  in  psychology  or  user  interface  design  is  desirable;  no  prior  knowledge  of  programming  or  modeling  is 
required. 

Registration:  Email  lisa@acm.org  to  be  registered  and  receive  the  seminar  url. 

Instructor: 

Professor  David  E.  Kieras 

Electrical  Engineering  and  Computer  Science  Department 
University  of  Michigan 

138  Advanced  Technology  Laboratory  Building 
1101  Beal  Avenue,  Ann  Arbor,  Ml  48109-2110 
Email:  kieras@eecs.umich.edu 
Phone:  (734)  763-6739 
Office  Hours:  1 1 :00  -  2:00  MWF 


David  Kieras  is  a  Professor  in  the  Electrical  Engineering  and  Computer  Science  and  Psychology  Departments  at  the 
University  of  Michigan.  His  primary  general  research  field  is  applied  and  theoretical  cognitive  psychology,  with 
specific  interests  in  human-computer  interaction,  cognitive  simulation  modeling,  human  performance,  and  natural 
language  processing.  His  research  has  been  supported  by  ONR,  NASA,  IBM,  and  DARPA.  He  has  presented  many 
tutorials  and  workshops  to  academic  and  industrial  audiences  on  human  performance  modeling  and  user  interface 
design. 


Guest  lecturer: 

Professor  Bonnie  John 
Human-Computer  Interaction  Institute 
Carnegie  Mellon  University 
3521  Newell-Simon  Hall 
5000  Forbes  Avenue 
Pittsburgh,  PA  15213 
Email:  bei@cs.cmu.edu 
Phone:  (41 2)  268-7182 

268-8004 


Bonnie  John  is  a  Professor  in  the  Human-Computer  Interaction  Institute  at  Carnegie  Mellon  University.  Her  primary 
research  interest  is  in  techniques  to  improve  the  design  of  computer  systems  with  respect  to  their  usefulness  and 
usability.  Much  of  her  work  focuses  on  cognitive  modeling,  where  she  works  within  a  unified  theory  of  cognition  to 
develop  models  of  human  performance  that  produce  quantitative  predictions  of  performance  with  less  effort  than 
prototyping  and  user  testing.  Her  research  has  been  supported  by  ONR,  NASA,  DARPA,  NSF,  Xerox,  GM,  and 
Boeing.  She  has  presented  many  tutorials  and  workshops  to  academic  and  industrial  audiences  on  human 
performance  modeling,  general  HCI,  and  usability  and  software  architecture. 

Seminar  moderator: 

Lisa  Neal 

Email:  lisa@acm.org 
Phone:  (781)861-7373 
Office  Location: 

3  Valley  Road 

02421 


Secretary: 


Sponsorship:  This  on-line  seminar  is  supported  by  a  grant  from  the  Cognitive  Science  program  of  the  Office  of  Naval 
Research.  Basic  research  on  computational  modeling  of  human  cognitive  architecture  has  been  a  major  emphasis  of 
ONR's  Cognitive  Science  program.  These  theories  have  now  reached  a  sufficient  level  of  maturity  that  many 
practical  applications  in  the  design  of  human  system  interaction,  as  well  as  training  applications,  are  now  feasible. 
This  seminar  is  intended  to  inform  participants  of  these  new  developments.  It  is  intended,  first  of  all,  for  personnel  in 
Navy  laboratories  who  are  involved  in  the  design  of  systems  that  will  have  human  users  or  operators,  and  for 
program  managers  planning  to  purchase  such  systems.  For  that  reason,  the  seminar  is  using  a  rather  low  tech 
approach  compatible  with  the  restrictions  of  the  Navy  Marine  Corps  IntraNet  (NMCI).  The  capacity  of  the  seminar  is 
large,  but  limited,  so  members  of  the  intended  audience  are  urged  to  register  promptly.  Registration  will  be  opened  to 
the  larger  public. 
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Appendix  B.  Survey  results 

Question  1 


1 .  How  did  you  hear  about  this  seminar? 


Cognitive  Science  Society  email 
announcement 


CTA  (Cognitive  Task  Analysis) 
email  announcement 


CHI  email  announcement 


Forwarded  email  announcement 


Email  from  Dr.  Susan  Chipman 


Individual  email  or  word  of  mouth 


KTH] 


Other  (please  specify) 


Response  I  Response 
Percent  I  Total 


(skipped  this  question) 


How  did  you  hear  about  this  seminar? 


Why  were  you  interested  in  taking  this  seminar? 


I  am  studying  human  factors  engineering  at  the  University  of  Virginia. 
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interest  in  user  experience  design 


This  is  my  field  of  study  (Human  Factors)  and  thus  it  is  always  interesting  to  broaden  one’s  perspective 


4.  I  am  working  with  GOMS  models  for  my  dissertation,  and  was  interested  in  upgrading  that  model  to  a  higher- 
level  architecture  -  Bonnie  John's  talk  in  particular  gave  me  the  insight  I  needed  to  leam  how  to  do  that. 


5.  Because  I'm  working  on  a  investigation  group  that  actually  is  carrying  out  a  proyect  that  involve  cognitive 
modelling 


6.  I  wanted  to  know  more  about  how  we  might  use  cognitive  modeling  in  Ul  design  and  what  the  state  of  the  art 
is. 


Useful  to  me  as  both  a  user  interface  designer  and  human  factors  researcher. 


HCI/Cog  Engineering  researcher  and  practioner.  Want  to  learn  more  about  other  related  disciplines. 


reguested  by  manager 


As  l/o  Psy  intern:  get  a  better  understanding  of  the  field 


Thopught  it  might  help  with  current  research 


My  research  lies  in  the  field  of  cognitive  modeling  in  medical  informatics 


I  am  an  educator  in  Computer  Science. 


The  content  is  related  with  my  Ph.D.  thesis. 


I  teach  user  interface  design  to  third  year  Computer  science  students  part  of  a  multmedia  design  course 


Research  on  Cognitive  Architectures 


interested  in  this  research  area 


I  am  working  in  this  area. 


because  i  am  doing  project  in  this  field. 


I'm  a  human  factors  graduate  student  and  have  interests  in  cognitive  issues  involved  in  video  games.  So, 
cognitive  architectures  seemed  like  a  relevant  topic. 


To  see  where  things  have  gone  in  the  last  20  years 


subject  matter 


The  topics  were  interesting  and  are  related  to  my  field. 


I  am  interested  in  Cognitive  modeling  in  general,  and  in  any  tool  to  make  it  faster,  in  specific. 


Interested  in  topic  and  speakers  are  well-known  experts 


I  do  some  interface  design  and  this  sounded  interestin 


My  science  work  is  related  with  field  of  this  seminar 


I  am  a  cognitive  psychologist  who  works  as  a  human  factors  engineer.  I  have  studied  this  topic  and  worked 
with  some  of  the  researchers  who  were  cited.  I  am  interested  in  using  these  tools  in  the  future. 


I've  been  substantially  involved  in  cognitive  modeling  in  the  past,  and  I'm  currently  using  EPIC  to  model  a 
dual  task  involving  auditory  information 


I'm  studying  human  factors  and  cognitive  science  at  the  University  of  Toronto,  and  so  this  discussion  spoke 
to  both  of  my  interests. 


I'm  a  first  year  PHD  student  and  I'm  interested  in  HCI. 


9 


I  am  currently  applying  NGOMSL  in  a  project  I'm  doing  at  SoarTech,  and  I'm  curious  about  how  architectures 
can  be  used  to  improve  the  validity  of  the  process.  I  also  wanted  to  ask  the  question  I  posed  to  Bonnie  about 
accounting  for  errors  in  vour  model. 


in 


To  learn  a  bit  more  about  cognitive  modeling  and  become  familiar  with  the  work  that  ONR  is  involved  with. 


thought  this  seminar  might  be  relevant  to  research  on  team  cognition 


Wanted  to  learn  more  about  cognitive  modelin 


H 

I 


As  a  cognitive  science  student  with  special  interests  in  human  computer  interaction,  I  was  curious  how 
cogsci  methods  might  contribute  to  better  interface  design. 


To  compare  cog  architectures. 


I  am  interested  in  HCI  nad  cognitive  studies 


Catch  up  on  current  status  &  maintain  awareness  of  field 


Because  I’m  considering  applying  this  approach  to  my  future  projects  about  risk  modellin 


To  get  an  idea  of  available  tools  for  interface  design  esp.  state-of-the-art  tools 


My  research  combines  CSCW,  distributed  team  work,  advanced  learning  technologies,  and  organizational 
communication  and  behavior.  A  particular  interest  is  in  modeling  distributed  cognition  to  better  understand 
interaction  in  distributed  groups.  I  am  presently  engaged  in  research  and  development  of  technologies  to 
support  distributed  real-time  adaptive  multi-agent  systems. 


I  do  &  supervise  software  development  in  a  niche  market  and  have  a  Ph.D.  in  cognitive  science,  so  I  do  pay 
attention  to  usability  but  mostly  at  the  level  of  functionality  (plus  some  general  principles  for  the  Ul).  We  can’t 
easily  conduct  usability  studies,  so  I  was  interested  in  what  the  modeling  options  are,  especially  those  that 
are  low-cost,  because  we're  a  small  company  and  because  my  time  as  a  programmer  is  actually  guite 
valuable. 


I  would  like  to  acguire  a  new  research  skill. 


Some  aspects  of  human  performance  are  pertinent  to  my  studies. 


I  want  to  expand  on  my  HCI  knowledge  in  general.  I'm  looking  fortechnigues  in  Ul  design  and  evaluation. 


I  am  a  computer  scientist  with  a  job  in  usabili 


My  degree  is  in  Cognitive  Science  /  HCI,  and  it's  a  continuing  interest  in  my  work  as  an  application 
developer. 


I  am  an  undergraduate  student  in  Computer  Graphics  who  is  participating  in  a  Flight  Display  Technology 
Human  Factors  Group  at  Purdue  University.  I  have  been  tasked  with  doing  literature  reviews  and  research 
on  cognitive  mental  modeling  and  related  areas  to  better  understand  what  a  pilots  cognitive  process  in  the 
cockpit  is  and  how  to  possibly  improve  cockpit  display  technology  to  improve  safety,  training  methods  and 
factors  to  consider  in  future  design  technology.  Also,  as  aviation  transitions  more  into  cockpit  displays,  what 
are  the  issues,  and  what  can  we  do  about  that? 


Sounds  guite  interesting  and  applicable  to  my  work  as  a  user  experience  architect. 


I  am  a  grad  student  researching  modeling  of  visual  search  for  application  to  HCI. 


et  survey/high-level  view  of  CM  tools  and  application  for  Ul  design 


Brush  up  on  modeling  knowledge.  See  how  it  can  be  applied  -  rather  than  just  hearing  about  research  into 
designing  architecture  (e.g.,  tuning  parameters) 


To  find  out  the  latest  thinking  about  modeling  and  user  interface  design. 


it  seemed  like  it  would  be  interesting— maybe  could  apply  to  current  work. 


It  is  related  to  my  graduate  work  in  modeling  human  computer  interaction. 


Cognitive  modeling  aspects 


I’m  from  China  mainland.  There  are  no  such  similar  courses  here  in  China.  CHI  is  very  important  in  the 
development  of  Chinese  Information  Technology,  and  will  be  more  important.  Cognitive  model  is  the  very 


base  of  interface  design,  so,  I  am  very  interested  in  it.  By  the  way,  I'm  working  in  a  HCI  Lab  in  the  Chinese 
Academy  of  Sciences.  I  learned  a  lot  from  it.  Thanks  a  lot. 


Interested  in  modeling  tools,  their  use,  how  best  to  start  the  modeling  process,  reliability,  usability.  Also  want 
to  know  what  others  in  the  field  are  doing 


I  m  looking  for  more  rigorous  interface  evaluation  tools.  Also,  I'm  embarking  upon  a  more  general 
investigation  of  applied  cognitive  psychology  in  hopes  of  finding  a  graduate  program  or  question  that 
addresses  my  interests.  I'm  not  interested  in  designing  specialized  jet  or  bullet  train  cockpits.  Instead,  I  am 
most  interested  in  "public"  interfaces  and  how  best  to  test  and  streamline  their  functionality.  Sending 
simulated  travelers  through  a  proposed  transit  system  or  network  would  be  very  helpful  in  identifying 
problems  system-wide. 


Doing  Ul  research  and  software  development. 


Involved  in  HCI  and  Cognitive  Engineering  at  work. 


I'm  an  interdisciplinary  student  in  Computer  Sciences  and  Social  Sciences.  I’m  about  to  enroll  the  Human 
Technology  Interaction  Msc  study  at  the  Eindhoven  Technical  University.  My  main  interest  is  User  Interface 
Design.  This  seminar  seemed  (and  was)  a  good  introduction  into  a  relatively  new  field  of  research. 


Because  I've  heard  peripherally  about  all  of  these  models  but  never  knew  where  to  start  in  terms  of  trying 
some  of  them  or  knowing  the  benefits/disadvantages  of  each. 


I  have  a  Ph.  D.  in  Mathematics  and  I  am  working  on  a  Ph.  D.  in  Educational  Psychology  and  a  Master  in 
Computer  and  Information  Systems.  I  found  this  seminar  to  be  very  useful. 


I  work  in  the  human  factors  field.  Sounded  like  an  interesting  topic. 


I'm  interested  in  learning  new  methods  for  Ul  testing  and  design. 


As  a  refresher  as  well  as  to  preview  it  for  some  usability  specialists  who  I  am  mentorin 


96.  I  I  don't  know  enough  about  this  topic.  I  am  developing  graduate  HCI  courses  for  a  degree  and  consider  a 
I  course  in  this  area  a  possibility. 


I  wanted  to  get  an  up-to-date  sense  of  the  state-of-the-art  of  cognitive  modeling  as  it  applies  to  interface 
design  and  evaluation. 


I  want  a  review  of  GOMS  and  other  modeling  methods 


Question  3 


3.  How  well  did  the  seminar  meet  your  expectations? 


■  US.'."'!;  ■  7  ,  i .. 

u  Exceeded, 

;  expectations 

Response 

Average 

1-5 

4%  (4) 

13%  (13) 

33%  (34) 

33%  (34) 

17%  (17) 

3.46 

Total  Respondents 


(skipped  this  question) 


Question  4 


Please  explain  your  answer  to  question  3: 


JL  I  The  content  was  good  and  the  seminar  lay-out  was  effective. 


1? 


3.  Very  impressed  with  the  content  of  the  seminar  and  how  easily  and  quickly  problems  were  troubleshooted.  I 
had  initial  problems  getting  the  audio  stream  the  2nd  day,  but  that  problem  was  easily  fixed  after  directing  my 
concern  to  the  bulletin  board  for  discussion. 


I'm  happy  with  the  issues  that  I’d  learned 


5.  well  that  was  an  "average"  of  several  factors.  I  was  pleased  with  the  format  of  the  seminar  and  happy  to  be 
able  to  attend  without  traveling  and  still  be  able  to  "chat"  with  other  attendees.  The  content  was  good  and 
met  my  expectations  (but  I  probably  had  fairly  high  expectations).  The  delivery  could  have  been  a  bit  livelier 
in  spots,  but  I  wouldn't  have  wanted  to  trade  content  for  "entertainment"  so  for  me  the  somewhat  dry 
academic  style  was  OK.  There  were  some  technical  glitches,  but  this  is  probably  inevitable  as  we  try  out 
these  technologies  and  learn  how  to  improve  things.  I  think  it  was  a  great  step  in  the  right  direction  and  I'm 
qlad  I  attended. 


Relatively  easy  interactions  with  the  presenters.  Presentations,  with  the  slides,  were  simple  to  follow. _ 


no  opinion 


The  topic  was  interesting.  However,  the  verbal  presentation  did  not  add  much  to  what  was  written  in  slides 


Thought  it  would  be  more  of  a  tutorial  on  interface  design  rather  than  building  models  to  represent  your  users 


I  was  frustrated  with  a  small  mistake  that  took  me  away  from  the  seminar  for  about  one  hour.. .a  little  tag  on 
the  seminar  URL  did  that. 


I  am  overseas  so  I  only  got  to  see  the  material  which  really  need  some  discussion.  I  could  not  get  the  audio 
to  work  at  all.  Also  the  time  this  was  going  on  was  not  a  good  time  for  me 


It  was  very  well  done  overall.  Thank  you  for  providing  this  service. 


Excellent,  interesting  and  informative 


Actually  i  m  looking  for  a  seminar  related  to  my  topic:  D  explicit  cues  to  Mutual  knowledge  enhance 
communication  on  virtual  whiteboard? 


15.  I  never  took  part  in  an  online  seminar  before,  so  I  didn't  really  have  any  expectations.  But,  the  seminar  was 
very  informative  and  well-run. 


I  couldn't  attend,  too  many  hoops  Lisa  was  helpful-  but  the  platform  was  too  complex. 


Presentations  from  the  first  two  days  were  good,  I  didn’t  attend  the  third  day.  The  format  was  more  like 
summarizing  the  concepts,  it  would  be  more  interesting  to  hear  about  new  things  that  we  haven't  heard 
before. 


I  only  heard  the  third  session  because  I  did  not  know  about  it  earlier. 


II 


Many  online  seminars  are  thinly  disguised  marketing  ploys.  But  this  one  had  a  lot  of  substance  and  made 
valuable  information  available. 


There  was  no  video.  Really  all  it  was  looking  at  a  powerpoint  while  the  facilitator  read  the  slides.  This  made  it 
rather  un-engaging.  I  was  under  the  impression  that  there  was  going  to  be  video  and  audio.  Not  really  sure 
what  was  added  by  doing  chat  online  when  questions  could  be  asked  over  the  phone.  It  essentially  was  a 
phone  conference. 


I've  expected  some  modeling  material  -  and  I've  got  it. 


I  was  extremely  pleased  with  the  level  of  detail  for  both  the  research  reviews  and  for  the  description  of  the 
available  tools.  Both  presenters  did  a  fantastic  job!!! 


I  know  the  field  relatively  well.  David  is  tops  and  I  was  delighted  by  Bonnie's  portion  of  the  seminar. 


I  could  not  attend  all  of  the  sessions,  however  I  found  it  very  well  organized.  Lisa  Neal  was  quick  to  respond 
to  e-mails,  and  I  appreciate  the  archives. ..effort  to  make  the  information  accessible  to  as  many  people  as 
possible.  _ 


Dr.  Kieras  and  Johns  were  clear  and  concise  in  delivering  the  seminar  AND  answering  questions.  They  were 
also  very  open  when  they  had  no  good  answer  for  a  question. 


n 


I 


I 


Will  need  to  check  for  more  detail  about  other  work  related  to  inexpensive  cog  modeling  for  design. 


Info  on  Cognitive  processes  was  very  good 


I  was  hoping  to  tie  this  info  more  into  what  the  Navy  is  doing  to  institute  Human  performance  into  it  schema. 


It  was  difficult  to  follow  for  me  online  due  to  time  constraints  and  technical  problems. 


It  was  both  broader  and  more  in-depth  than  I  thought  it  would  be.  The  leaders  were  very  good  about  making 
sure  that  participant's  questions  were  answered. 


I'm  a  neophyte  in  this  subject.  The  seminar  defined  cognitive  modeling,  described  how  it  can  be  used  to 
evaluate  interfaces,  provided  practical  examples  of  its  application,  and  then  discussed  current  limitations.  I 
would  say  that  it  exceeded  expectations,  but  my  expectations  were  high  going  into  it. 


It  would  have  been  great  to  see  more  of  Bonnie  John's  work  and  less  of  David's  intro. 


didn't  think  that  a  web-based  presentation  could  be  that  effective 


Because  I  am  not  directly  involved  in  user  interface  design  etc  I  only  expected  parts  of  the  seminars  e.g.  on 
practical  issues,  ACT-R  and  other  architectures  to  be  of  direct  use. 


I  was  looking  for  a  useful  overview  and  was  satisfied  with  the  product. 


For  me,  it  completely  missed  the  mark.  The  'model  human  processor1  implemented  as  a  set  of  production 
rules  may  be  descriptive,  but  it's  too  complex  and  time  consuming  to  be  useful  for  everyday  design. 


I  didn't  realize  this  seminar  would  dive  in  to  cognitive  modeling  simulation,  I  thought  it  would  be  more  about 
cognitive  modeling  techniques,  although  I  clearly  recall  the  disclaimer  about  no  hands  on  cognitive  modeling 


copies  of  material  will  provide  valuable  future  reference.  Hearing  the  seminar  provided  clarification  of  the 
material. 


It  gave  me  an  excellent  overview  of  what  cognitive  modeling  is  all  about.  I  liked  the  idea  of  software  being 
made  available.  I  would  like  to  see  some  specific  examples,  in  software,  for  us  to  play  with. 


Since  I  had  just  read  a  few  of  Dr.  Kieras'  publications  about  GOMS  and  Cognitive  modeling,  I  wasn’t  sure  I 
would  get  much  new  information  out  of  the  seminar,  but  I  found  it  worthwhile.  I  especially  enjoyed  the  chat 
session.  It's  too  bad  the  technology  chose  to  mediate  that  session  didn't  permit  everyone  in  the  seminar  to  be 
logged  into  it  at  the  same  time. 


the  last  day  really  was  the  most  useful  by  describing  how  coqtool  speeds  development. 


I  was  unable  to  participate  as  much  as  I  wanted  too  but  the  content  is  well  prepared. 


Too  general,  the  examples  were  not  very  instructive 


I  found  the  presentation  was  very  professional  and  informative. 


It  was  superb! 


It  was  what  I  expected.  An  overview  of  the  different  modeling  methods  and  applications  examples.  The 
format  made  it  hard  for  me  to  follow  the  lectures  (I  constantly  got  lost  between  the  slides  and  lecture),  even 
though  the  presenters  did  GREAT  in  trying  to  minimize  this  by  their  reference  to  slide  numbers,  etc.  Also  I 
would  have  liked  to  hear  even  a  brief  discussion  on  industry  applications  rather  than  only  the  'researchy'  type 
discussions  (note  that  I  was  not  able  to  listen  to  session  #3,  due  to  schedule  constraints). 


My  initial  expectations  were  to  see  a  complete  modeling  of  a  problem  solving  process.  But  I  soon  found  I  was 
interested  in  the  research  methodology,  and  the  3rd  day  presented  real  insights  into  how  it  can  be  applied  in 
classroom  situations. 


I  wish  I  could  have  sat  in  more  of  it  but  I  heard  David  Kieras  speak  and  he  was  fantastic.  He  is  very  much  in 
touch  with  the  current  research  and  I  found  his  presentations  helpful. 


interpret  "cognitive  modeling"  as  being  more  closely  related  to  human  performance  than  artificial  intelligence 
and  computer  programmin 
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I  was  not  able  to  attend  all  the  sessions  due  to  the  fact  that  some  of  my  classes  were  scheduled  at  the  same 
time.  I  felt  that  the  audio  files  were  a  big  help. 


The  introduction  to  the  cog.  architectures  was  fine  but  I  didn't  like  the  practical  part,  it  seemed  to  be  very 
narrowed.  Additionally  the  technical  realization  wasn't  that  good. 


I  did  not  know  what  to  expect. 


My  goals  were  met. 


Because  my  knowledge  about  user  interface  design  isn’t  enough  and  I  was  afraid  about  understand  the 
seminar,  but  the  lessons  were  so  clear  that  I  follow  them  easily 


I  found  the  chosen  topics  to  be  very  useful  and  informative. 


I  have  set  up,  managed,  and  taught  several  online  courses  for  students  at  both  the  undergraduate  and 
graduate  levels.  As  a  result,  I  am  familiar  with  a  variety  of  ways  to  design  courses,  present  material,  and 
encourage  interaction  to  create  an  effective  virtual  environment  for  work  and  learning.  This  seminar  was  well- 
designed,  provided  flexible  tools  for  reception  and  interaction,  and  was  well-presented.  The  few  technical 
glitches  that  occurred  were  quite  insignificant.  The  facilitators  and  presenters  did  a  great  job  of  pacing  the 
seminar  and  presenting  the  material  clearly.  The  Q  &  A  sessions  handled  queries  from  multiple  sources  very 
well.  Lisa  Neal  did  a  great  job  of  keeping  attendees  informed  and  managing  any  unexpected  technical 
qlitches. 


I  thought  this  might  be  a  very  academic  conference  with  a  lot  of  focus  on  research,  and  if  so,  I  planned  to 
leave  after  the  first  day  or  as  soon  as  it  got  too  abstract  to  apply.  Actually  it  got  more  and  more  useful  as  the 
conference  went  on.  I  plan  to  look  at  some  of  the  tools  mentioned  for  some  of  our  simpler  interfaces,  like  our 
web  product.  It  might  also  be  a  way  of  teaching  other  programmers  a  bit  more  about  usability.  The  seminar 
was  also  helpful  for  me  in  focusing  my  thoughts  on  what  aspects  of  usability  give  us  the  most  bang  for  the 
buck. 


The  seminar  met  expectations,  the  technical  difficulties  got  in  the  wa 


It  was  very  easy  to  attend.  I  liked  having  the  chat  as  a  separate  channel  to  the  voice  presentation.  It  made  an 
excellent  Q&A  forum,  and  you  don't  get  that  kind  of  timely  attention  from  a  stand  up  presentation.  As  far  as 
the  material  covered,  I  found  it  helpful. 


It  was  mostly  focused  on  performance,  as  though  users  are  robots.  Usability  should  be  user-oriented  until  a 
time  when  we  can  accurately  depict  a  real  person  with  a  simulation.  Other  than  that,  it  was  good  A  A 


|| 


The  access  was  amazingly  simple  and  pervasive  (text  chat,  phone,  audio)...  and  it  all  ‘worked!*  The  topic 
was  well-presented  by  both  presenters  (Kieras  and  John),  and  the  moderation  was  flawless  (Lisa  Neal).  Slide 
show  collateral  required  some  outside  assistance  with  a  mirror  site,  but  that  worked  well,  too. 


I  didn't  know  what  to  expect,  brand  new  to  this  area.  I  learned  a  lot  just  about  the  different  terms  and  have  a 
better  understanding  of  some  of  the  things  involved  in  this  area.  Kind  of  an  intro  for  me  as  I  hope  to  spend 
the  next  few  years  learning  more  in  these  areas.  Also,  got  good  references  and  great  notes  to  review. 


Overall,  it  was  much  more  technical  at  its  core  than  I  expected.  Bonnie’s  presentation  brought  the  practical 
application  to  light  though.. .1  was  getting  lost  almost  w/  the  level  of  technical  knowledge  needed  to  utilize 
cognitive  modeling,  then  Bonnie  cleared  it  up  for  me  w/  showing  the  practical  version  of  her  Cog  Tool.  I'm  a 
Ul  designer,  who  also  is  responsible  for  usability  engineering,  information  architecture  and  design,  and  over 
all  user  experience  planning.  I  see  the  benefits  in  doing  some  of  this,  but  until  I  saw  Cob  T ool,  was 
thinking. .how  would  I  go  about  this? _ 


A  wide  variety  of  topics  in  modeling  with  particular  emphasis  on  issues  in  usability  testing.  There  was  more 
material  on  practical  issues  with  modeling  than  I  had  expected. 


In  addition  to  survey  view,  gained  some  useful  insights  into  design/CTA  interface  and  sequencing,  and 
specifics  on  some  recent  developments  (Johns  Cogtool)  of  tremendous  interest.  The  further  reading  list  was 
also  very  valuable.  Course  provided  a  tremendous  amount  of  insights  and  information  for  a  modest  amount 
of  participants'  time. 


Great  ideas,  research,  and  results! 


■ 


I  listened  to  most  of  the  seminar  on  streaming  audio,  but  didn't  follow  along  too  closely  with  the  slides.  I  think 
the  seminar  was  a  success 


I  really  like  the  organization  of  the  material,  and  the  delivery  of  information,  it  is  a  successful  online  seminar. 


I  want  to  know  what's  going  on  with  the  Cognitive  model.  I  learned. 


This  was  one  of  my  first  telecon  seminars,  so  I  wasn't  sure  what  to  expect.  There  were  some  bugs  that 
needed  ironing,  and  the  snow  on  Monday  kept  us  out,  but  generally  I  felt  that  it  was  good.  The  Q&A  was 
really  a  positive,  and  the  chance  to  have  questions  asked  answered  in  the  chat  was  really  a  good  thing. 
Having  two  instructors,  plus  Lisa,  meant  that  while  one  was  talking,  somebody  else  was  available  to  monitor 
chat  and  answer  questions  as  we  went. 


The  28  Feb  session  was  a  bit  more  technical  than  I  had  hoped  and  I  wondered  if  I  had  made  a  mistake  in 
signing  up.  But  the  1  March  session  offered  more  pragmatic  examples  which  made  it  easier  for  me  to 
understand  what  Dr.  Kieras  was  describing.  I  am  now  looking  forward  to  Dr.  John's  presentation  on  2  March. 


I  wasn't  able  to  attend  the  second  seminar  but  the  first  seminar  mostly  had  Kieras  reading  aloud  from  his 
PowerPoint  slides.  So  I  logged  off  and  read  the  slides  on  my  own. 


Average.  Presentations  were  informative  but  rather  dry.  I  could've  read  the  PowerPoint  slides  and  been  ok. 


The  seminar  was  comprehensive,  interactive,  educative  and  fun.  I  was  pleased  to  be  able  to  ask  questions. 
Although  I  hoped  that  the  models  would  be  more  capable,  I  found  out  that  there's  a  wide  open  field  of 
possible  future  research.  I'll  surely  take  some  time  for  a  hands-on  experience  with  CoqTool  in  the  near  future! 


This  was  exactly  what  I  was  looking  for  in  terms  of  getting  an  introduction  to  this  topic. 


77.  I  expected  more  explanation  and  elaboration  than  a  basic  reading  of  the  slides.  The  Q&A  part  was  helpful, 
however 


I  really  like  the  content,  however,  I'm  a  bit  disappointed  in  the  lack  of  depth  of  concepts.  Also,  it  wasn't  clear 
to  me  what  Epic  stood  for. 


I  think  that  this  online  course  is  excellent.  The  only  problem  is  that  I  had  difficulty  following  the  last  example 
and  would  have  liked  the  instructor  to  have  gone  over  it  more  slowly. 


The  presentation  technology  worked  very  well.  The  content  material  was  excellent  and  well  presented.  The 
question  and  answers  were  invaluable  in  getting  a  sense  of  what  the  speakers  perspective  of  the  state-of- 
the-art  of  cognitive  modeling  -  what  it  is  good  for  and  what  it's  limitations  are. 


I  liked  the  screenshots  of  EPIC.  The  preface  said  there  would  be  no  "how-to"  and  I  was  disappointed.  Kieras' 
review  of  the  EPIC  screenshots  left  my  technical  side  satisfied. 


Question  S 


5.  Instructions  for  registering  and  attending  the  seminar 


(skipped  this  question) 


HiHBai 

Moderate  ; 

m 

Difficult  1 

Seminar  registration  was 

77%  (75) 

9%  (9) 

3%  (3) 

1%(1) 

1.42 

Attending  the  seminar  was 

57%  (54) 

12%  (11) 

6%  (6) 

3%  (3) 

1:77;'::'/; 
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Question  6 


6.  How  did  you  experience  the  seminar?  (check  all  that  apply) 


Responst 

Percent 


Streaming  audio  (sound  over  the 
computer) 


Archive  (recorded  playback  of  the 
seminar) 


If  you  used  more  than  one 
option,  please  say  why: 


Total  Respondents 


(skipped  this  question) 


How  did  you  experience  the  seminar?  (check  all  that  appl 


Because  my  connection  wasn't  optimum  and  I've  no  time  to  connect  all  the  time 


2.  I  had  some  problems  with  the  audio,  so  went  to  the  phone  for  part  of  it.  I  also  wanted  to  replay  some  from  the 
archive. 


Conflicts  with  work  commitments. 


listened  to  one  seminar  live,  other  two  seminars  through  archive 


5.  Time  restraints.  2.5  hours  for  three  days  takes  up  a  good  chunk  of  the  week.  Especially  right  after  lunch 
when  I  am  at  my  most  productive  state 


Had  time  constraints  for  the  second  day  to  attend  live. 


I  couldn't  attend  the  full  seminar  as  I  had  to  teach  classes. 


Phone  was  easiest,  but  I  couldn't  attend  every  da 


Actually,  I  just  read  the  slides 


I  have  downloaded  the  archived  audio  so  that  1  can  listen  to  it  again  later.  I'm  not  sure  that  I  will,  but  it  could 
be  an  excellent  resource. 


I  had  to  use  the  phone  for  a  bit  due  to  a  problem  with  the  streaming  audio  link;  in  addition  I  had  to  take  my 
office  mate's  schedule  into  account. 


I  missed  the  first  day,  so  I  used  the  archives  to  prepare  for  the  second  day.  I  also  used  the  streaming  audio 
to  listen,  then  called  into  the  voice  line  to  ask  questions.  This  left  my  phone  line  available. _ 


sometimes  I  could  not  get  streaming  audio  and  reverted  to  phone. 


Just  wanted  to  see  how  well  it  worked.  Preferred  the  phone. 


Well  some  difficulties  with  the  streaming  audio  meant  that  I  had  to  listen  to  the  recorded  playback 


I  missed  the  2nd  seminar  because  I  have  to  teach  and  have  to  access  via  stream  audio  because  I  have  no 
phone  access  for  the  3rd  seminar. 


Prior  commitments  meant  i  was  unable  to  stream  the  audio  at  the  time,  the  archive  was  a  great  idea-  as  it 
meant  I  could  access  all  three  sessions  the  next  day.  Thank  you! 


I  saw  the  slides  before  the  seminar  and  decided  not  to  attend. 


Mostly  streaming  audio.  However,  there  was  a  time  when  there  were  some  problems  with  the  stream  that  I 
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had  to  switch  over  to  phone.  I  much  preferred  streaminq  format. 


I  downloaded  the  slides  from  the  web  site  and  listened  on  the  telephone. 


streaming  audio  did  not  work  on  dav  3 


The  material  presented  by  Drs.  John  and  Kieras  was  complex  and  voluminous.  It  is  great  to  have  the  audio 
archive  to  replay  particularly  interesting  and/or  challenging  segments  of  presentations. 


I  downloaded  the  archive  (but  haven't  reviewed  it)  because  I  was  called  away  during  part  of  the  first  day  and  I 
want  to  go  back  and  see  what  I  missed. 


Some  portions  of  the  audio  were  hindered,  hampered  by  technical  difficulties;  the  archives  made  it  possible 
to  catch-up  those  parts  missed. 


Text  chat.  Very  useful  having  text  chat  running  at  the  same  time,  especially  in  a  distance  seminar  where 
asking  the  speaker  questions  during  the  talk  is  impossible.  Audio  archive  was  important  to  disseminate  the 
talk  to  other  in  my  company  who  were  unable  to  attend  in  person. 


On  Wednesday,  tried  to  get  on  streaming-  broken  link.  Was  able  to  go  back  to  streaming  after  break. 


I  started  w /  the  phone,  but  realized  that  the  streaming  audio  was  going  to  be  much  better  for  me.  I  sit  in  an 
open  space  and  don't  have  a  head  set,  so  using  my  headphones  I  can  listen  directly  to  the  computer. 


I  used  streaming  audio  the  first  two  days.  The  third  day  I  used  phone  due  to  troubles  with  streaming  audio 
the  first  two  days. 


attempted  to  use  streaminq  audio  without  success 


chat  was  useful  for  asynchronous  comments 


Cannot  meet  the  seminar  time  as  it  coincide  with  work 


We  had  a  group  that  totaled  about  10  who  were  interested  or  attended  part  of  the  seminar;  those  who  missed 
part  were  interested  in  reviewing  the  archive  (or  at  least  the  Q&A  part,  since  some  of  the  slide  presentation 
could  be  read  just  as  easily. ;)  We  didn't  even  try  streaming  audio,  but  I  know  folks  here  listen  to  radio 
stations,  etc,  so  I'm  sure  it  would  have  worked. 


Due  to  some  other  obligations,  I  wasn't  able  to  attend  the  first  1 5  minutes  of  every  meetm 


Question  7 


7.  Was  streaming  audio  an  option  for  you? 


Respo 

nse 

Percen 

:V:  i  V 


Yes 

1— — — — -j 

71.7% 

71 

10.1%  I  10 


18.2%  I  18 


Total  Respondents  I  99 


(skipped  this  question)  I  3 
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BIBBS 


Describe  anv  issues  you  had  with  sound  quality  and  the  duration  of  the  problem 


The  sound  lost  for  few  seconds  or  minutes,  but  after  that  time,  the  quality  was  fine. 


the  audio  was  fine  except  when  there  were  problems  like  it  stopping  toward  the  end  of  the  first  session. 


It  will  not  run 


It  was  great  when  I  could  hear  the  audio.  There  were  several  times  when  I  lost  audio,  but  it  could  be 
problems  on  my  end. 


One  of  the  speakers  seemed  to  be  far  away  from  the  microphone  (on  Monday) 


6.  I  only  used  the  telephone,  and  I  was  very  satisfied  with  that  audio.  I  though  the  moderators  did  a  really  good 
job. 


Streaming  audio  generally  has  a  warbly  quality.  This  was  no  exception. 


no  problems 


wm\ 


Quality  on  streaming  audio  was  better  than  phone  quality  for  Barbara  Tversky's  lecture. 


operator  interruption  5sec.  but  a  critical  introduction  time. 


every  once  in  a  while  the  feed  would  drop  out  -  may  have  been  on  my  end? 


I  could  not  access  to  the  stream  audio  at  the  beginning  of  the  3rd  seminar  because  the  link  sent  out  was 
broken. 


No  problems  at  all. 


On  the  first  day,  the  stream  seemed  to  stop  and  I  had  to  switch  over  to  the  phone.  I  later  switched  back.  I  do 
not  know  the  reason  for  the  problem.  I  was  glad  that  I  had  the  phone  as  a  backup. 


over  the  phone  the  audio  seemed  fine.  Sometimes  a  background  speaker  or  a  call-in  question  was  too  quiet. 


I  choose  the  phone  because  the  sound  quality  was  superior. 


No  problems  with  audio 


some  time  it  broke  up  and  there  was  silence  (on  day  1 


I  was  not  able  to  use  steaming  audio. 


Loss  of  sound  during  brief  moments. 


Voices  did  not  have  a  great  sense  of  immediacy.  In  day  one  or  two  there  was  some  sustained  coughing  on 
the  line  (hope  everyone  is  OK) 


IQ 


X 


23. 

The  only  problem  on  the  first  day  was  toward  the  end  of  the  seminar.  The  last  1 0  minutes  or  so,  the  sound 
cut  in  and  out  and  the  presentation  was  hard  to  follow. 

1  24.  1  1  hadn’t  any  problem,  but  my  system  is  very  old  j 

r 

There  were  some  occasions  when  the  audio  was  dropped  intermittently  for  periods  of  30  seconds  to  a  couple 
of  minutes  during  sessions  1  and  2. 1  hope  the  missing  portions  are  present  in  the  archive.  Otherwise,  the 
audio  was  of  excellent  quality. 

27. 

There  were  some  periods  of  missing  audio,  but  usually  for  only  a  few  minutes  at  a  time,  and  only  during  the 
space  of  a  half-hour  on  the  second  presentation  day.  Otherwise,  the  audio  quality  was  very  good  -  no 
obvious  "digitizing"  effects. 

r 

For  most  part,  audio  when  it  came  in  was  fine;  breakage  of  audio  occurred  several  times.  1  was  never 
completely  disconnected,  just  couldn't  get  on  Wednesday  at  1pm.  Glad  to  have  access  to  archives  for 
missed  material  and  1  was  impressed  with  the  communication  and  prompt  handling  of  the  issues. 

F 

Tuesday  session  had  some  hiccups. ..there  was  one  (1)  instance  of  the  audio  skipping,  that  is  repeating  a 
phrase  a  few  times. 

F 

When  the  audio  was  present,  which  was  most  of  the  time,  the  quality  was  good.  Audio  dropping  during  the  | 
first  day  and  not  reconnecting  after  the  break  on  the  second  day  was  problematic.  f 

F 

answer  to  8  refers  to  phone,  was  not  able  to  use  streaming  audio  due  to  lack  of  sw  support  (stock  Mac  OS  1 
X/Safari)  I 

Couldn't  get  streaming  audio  to  work  on  the  Macintosh.  Vivavid  said  that  the  conference  wasn't  set  up  for  | 

Macs.  1 

w 

the  sound  quality  was  an  issue  to  the  degree  that  the  voices  were  not  crisp,  there  was  a  background  of  low  | 
level  white  noise— probably  due  to  the  medium  1 

W 

We  were  only  on  the  phone.  It  worked  well  there,  occasionally  one  person  would  be  louder  or  quieter  than  | 

others,  but  it  was  fixed  by  moving  closer.  1 

0 

Sound  quality  itself  was  fine,  1  could  hear  words  clearly.  1  experienced  several  long  (20-30  seconds) 
moments  of  the  stream  dropping  completely.  Using  MS  Media  Player  and  it's  been  pretty  reliable  in  the  past. 

I  36.  1  Some  spots  in  the  audio  for  several  secs  now  and  then,  but  no  big  problems.  1 

1  38.  |  There  were  some  lapses  of  sound  -  some  up  to  ~  15  seconds.  1 

Unfortunately,  the  audio  cut  out  at  some  important  parts  of  the  example,  making  it  difficult  to  follow.  It  is 
extremely  important  for  the  instructor  to  let  the  audience  know  what  page  they  are  on. 

0 

Sound  was  not  always  very  clear  and  sometimes  faded  in  and  out  --  but  overall  1  could  hear  and  understand 
the  speaker  just  fine. 

Question  10 


?n 
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I  missed  2  seminar  and  had  to  listen  t  the  archive. 


Accessing  the  archives  at  a  later  date  did  not  enable  me  to. 


Did  not  attend. 


didn't  have  any  questions  -  wasn't  quite  the  subject  matter  I  thouqht. 


Did  not  have  the  urge.  The  conference  technology  was  not  the  barrier. 


Many  good  questions  were  already  asked  in  the  sessions. 


just  checking  things  out.  if  in  the  course  of  applying  the  info,  I’ll  contact  the  presenters. 


Unprepared 


I  was  only  able  to  listen  to  portions  of  the  presentation  and  did  not  attend  the  QA  sessions. 


Didn't  have  any  to  ask. 


I  used  the  archived  version 


34.  I  was  already  familiar  with  the  topic.  Also  the  content  in  the  slides  answered  must  of  my  questions  (i.e.  where 
to  qet  resources  for  using  this  models  in  industry). 


Waaay  out  of  my  research  area. 


I  was  more  interested  in  hearing  what  others  thoughts  were. 


No  pressing  questions. 


Didn't  have  any  that  were  not  asked  by  someone  else. 


Because  I  understand  everything  (or  I  think  so 


Not  enough  knowledge  base  to  interact  and  understand  all  that  was  being  asked.  The  questions  I  do  have 
pertain  to  approaching  this  area  in  regards  to  others  who  are  working  in  modeling  in  Cockpit  display/  flight 
applications.  Thouqht  I  could  send  personal  email  later. 


The  Yahoo  chat  did  not  work  for  me. 


Couldn't  access  the  text  chat.  Maybe  it  was  a  java  problem  on  my  end.  Yahoo  has  no  real  online  support 
documentation  for  the  chat  tool. 


topics  were  covered  by  presentation  and  other  questions 


I  was  working  while  listening  to  the  seminar,  so  I  was  only  half-involved  with  the  material. 


I  was  more  interested  about  the  overview,  and  didn't  feel  the  need  to  ask  questions. 


Too  late  since  I  only  can  listen  to  the  archive. 


I  could  not  participate  in  the  seminar  because  of  the  time  difference. 


Streaming  audio. 


I  couldn't  get  the  drift  of  the  topic  -  not  even  enough  to  ask  questions. 


no  pude  conectarme  al  chat.  La  red  de  la  universidad  no  tenia  acctivado  el  servicio 


It  seemed  like  the  queue  was  full 


The  audience  questions  were  excellent. 


Difficult  to  do  on  streaming  audio 


?? 


1 1 .  For  each  instructor,  please  provide  feedback  on  his  or  her  presentation: 


David  Kieras:  Rate  the  content  of 
the  presentation 


David  Kieras:  Rate  the  quality  of 
presentation  materials  (slides,  etc.) 


David  Kieras:  Rate  the 
organization/flow  of  the 
presentation 


David  Kieras:  Rate  the  speaker’s 
on-line  presentation  skills 


David  Kieras:  Rate  the  level  of 
interactivity 


Bonnie  John:  Rate  the  content  of 
the  presentation 


Bonnie  John:  Rate  the  quality  of 
presentation  materials  (slides,  etc.) 


Bonnie  John:  Rate  the 
organization/flow  of  the 
presentation 


Bonnie  John:  Rate  the  speaker’s 
on-line  presentation  skills 


Bonnie  John:  Rate  the  level  of 
interactivity 


Poor 

Fair 

Very  good 

1  Excellent 

1%(1) 

0%  (0) 

I  28%  (26) 

39%  (36) 

32%  (29) 

0%  (0) 

9%  (8) 

30%  (27) 

35%  (32) 

26%  (24) 

0%  (0) 

8%  (7) 

22%  (20) 

42%  (38) 

29%  (26) 

4%  (4) 

12%  (11) 

30%  (27) 

25%  (22) 

28%  (25) 

3%  (3) 

17%  (15) 

28%  (25) 

23%  (21) 

29%  (26) 

0%  (0) 

1%(1) 

19%  (15) 

40%  (31) 

39%  (30) 

0%  (0) 

5%  (4) 

17%  (13) 

45%  (35) 

32%  (25) 

0%  (0) 

1%(1) 

19%  (15) 

42%  (32) 

38%  (29) 

0%  (0) 

0%  (0) 

25%  (18) 

43%  (31) 

32%  (23) 

1  %  (1 ) 

8%  (6) 

33%  (24) 

21%  (15) 

37%  (27) 

Total  Respondents  I  94 


(skipped  this  question)  I  8 


Question  12 


12.  Would  you  attend  a  similar  on-line  seminar  in  the  future? 


No, 

definitely 

not 


Maybe 


Yes,  Response 
definitely  Average 


Attend  again 

1%  (1) 

0%  (0) 

16%  (16) 

24%  (24) 

60%  (61) 

4.41 

,  ;  Total  Respondents  I  102 


(skipped  this  question) 


Question  13 


If  so,  what  topics  or  instructors  would  interest  you  most? 


More  issues  related  to  cognitive  modeling  -  perhaps  even  learning  the  architecture  by  means  of  a  tutorial. 


Evolutionary  Computation  Learning  Machine  Artificial  Intelligence 


3.  I'm  really  short  of  time  right  now  and  you  are  threatening  to  close  this  tomorrow.  Could  I  email  this  to  you 
later,  Lisa? 


More  of  the  same,  experts  in  related  design  fields. 


I  would  like  to  discuss:  Sense  making,  LSA. 


6.  Cognitive  science  VR  Instructional  Design  E-leaming  User  interface  for  web  development  USABILITY 
*********** 


7.  I  Anything  that  has  to  do  with  applying  psychological  principles  to  technology.  Maybe  an  artificial  intelligence 
I  course? 


8.  Video  games,  the  cognitive  mechanisms  at  work  when  they  are  played.  Neuroergonomics  Multimodal(ity) 
Interfaces  Adaptive  Automation 


9.  I  volunteered  a  seminar  topic  (subject  matter  expert)  to  CTA/Aptima  and  never  got  a  reply.  Pretty 
unorganized  and  lame  if  you  ask  me. 


Anything  related  to  cognitive  science,  cognitive  systems  engineering,  Al. 


Cognitive  task  analysis  Acguisition  of  expertise  Intelligent  tutoring  systems  Scenario-based  trainin 


modeling  of  group-related  behaviour  and  group  status  issues 


Discourse  models,  Intelligent  user  interfaces,  Human-robot  interaction 


Knowledge  Management  and  Human  Performance 


HCI  and  cognitive  modellin 


"Magic  and  interface  design"  by  Bruce  Tognazzini : 


any  instructional  technology  topic,  particularly  those  having  to  do  with  distance  learnin 


Activity  Theory  or  Distributed  Cognition  or  Ethnographic  Methods  or  Abowd's  Ubicomp  work 


Cognitive  modeling,  User  Centered  Design,  Human  Factors  Engineering,  Cognitive  Science,  Visual 
Perception,  Attention. 


more  on  how  to  do  cognitive  modelin 


Not  sure-  my  specific  interest  is  in  ACT-R  so  uses  of  this  architecture  would  be  useful. 


Ul  design  related,  especially  enhancing  usabili 


I  like  the  advanced  HCI  type  of  topics  and  hearing  about  research  areas  -  sorta  like  going  to  a  conference 
presentation,  but  staying  home! 


Susan  Chipman  -Knowledge  Acguisition  -or-  how  to  take  Cognitive  development  and  learning  into  account  in 
your  design  and  cognitive  task  analysis. 


more  on  handheld  interface  development  issues. 


More  cognitive  science... 


I  am  interested  in  similar  topics  on  Cognitive  Modeling  and  methodologies. 


e-leaming,  Distance  Education  issues  and  related  topics. 


Anything  related  to  HCI  and  theory  applications,  rather  than  only  current  research  topics.  I  am  all  for 
matching  the  research  to  industry  applications. 


-  applications  of  cognitive  modeling  in  the  classroom 
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31.  |  Current  applied  research  in  cognitive  and  human  performance  modeling.  I  did  not  have  a  chance  to  hear 
Bonnie  John  speak  so  would  be  interested  in  hearing  her  in  the  future. _  ^ 


I  would  like  to  hear  Bonnie  John's  presentation  because  I  missed  it. 


34. 


Anything  that  elucidates  on  cognition,  perception,  emotion  and  how  these  aspects  interact  and  affect  a 
person’s  ability  to  interact  and  communicate.  _ _ 


35.  I  I  don’t.  Maybe,  I'll  take  more  time  in  emotions  and  Keates  looks  interesting 


36. 


There  has  been  a  lot  of  work  in  the  intelligent  user  interface  area  but  I’m  not  sure  if  most  people  who  attend 
CHI  also  attend  IUI  for  example.  It  would  be  nice  to  have  speakers  that  are  from  both  groups. _ 


37.  I  Distributed  Cognition,  Contextual  Design,  Information  Visualization,  Multi-Agent  Systems,  Web  Ontology 
Languages _ _ _ _ _ 


38.  I  Topics  relating  to  software  development,  testing,  usability,  etc. 


41.  I  New  methods  of  traditional  usability  I 


42. 


43. 


44. 


45. 


-Use  Case  modeling  -Requirements  gathering  and  management  -Cost-justifying  usability  methods  • 

More  on  cognitive  mental  modeling.  Applications  to  military  and  flight  simulation,  air  traffic  control,  cockpit 
display,  etc... 

More  related  on  HCI,  usability,  interaction  design.  Practical  application  of  user-centered  design  methods  and 
theory. 

Modeling  of  visual  search:  Mike  Byrne,  Anthony  Hornof,  Jeremy  Wolfe  Integrating  models  of  complementary 
or  competing  tasks:  Dario  Salvucci,  ??? 


|46ijBonnieJohns^av^ 

|  More  into  modeling  human  computer  interaction.  Maybe  topics  related  to  human  in  the  loop  systems. 

[  Cognitive  model  for  Ubiquitous  computing  Distributed  Cognition  I 


50.  |  It  was  good  to  get  some  general  instruction  /  guidance  on  tools.  That  might  be  helpful.  Having  a  particular 
customer  base  present  would  be  nice  from  my  perspective  (like  the  Navy  and  the  tools  they  use;  could  look 
at  a  particular  program  perhaps,  such  as  Aegis,  Tomahawk,  DDX,  Subs,  etc)  Information  presentation  and/or 
decision  making.  Tools,  techniques,  etc.  Training  issues,  howto  make  training  part  of  the  system  as  it  is 
used,  not  an  afterthought  (or,  how  to  design  the  system  with  training  in  mind)  -  Dr.  James  Pharmer,  NAVAIR 
Orlando,  is  one  name  that  pops  to  mind. 

Topics:  Applied  cognitive  psychology  Cross  cultural  interface  design  Instructors:  Richard  Nisbett  Jef  Raskin 
Don  Norman 


about  HCI  is  okay  with  me! 
Not  sure  right  now 


topic  in  the  broad  spectrum  of  human  factors. 


This  is  an  excellent  way  to  keep  up  with  the  state-of-the-art  in  Cognitive  Psychology/Cognitive 
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Enaineerinq/HCI  without  the  need  to  travel  to  conferences.  It  is  low  cost  in  terms  of  time  commitment. 


End-User)  Programming  by  Demonstration  tools  and  research,  i.e.  Alan  Cipher  et  al 


14.  This  seminar  was  paid  for  by  the  Office  of  Naval  Research.  Would  you  be  willing  to  pay  for  this  type  of 
seminar  in  the  future  (check  all  that  apply): 


Respondent 

Total 


1  would  be  willing  to  pay  for  the 
phone  call: 

17%  (16) 

37%  (35) 

40%  (38) 

11%  (10) 

^>■95^:' 

1  would  be  willing  to  pay  a  fee  for 
the  seminar: 

6%  (6) 

28%  (27) 

58%  (56) 

11%  (11) 
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My  employer  would  be  willing  to 
pay  for  the  phone  call: 

16%  (16) 

21%  (20) 

29%  (28) 

37%  (36) 

97 

My  employer  would  be  willing  to 
pay  for  the  seminar  fee: 

9%  (9) 

20%  (19) 

38%  (37) 

36%  (35) 

.'v|:  97 

(skipped  this  question) 


Question  15 


15.  Indicate  your  primary  business  sector: 


Respons 

e 

Percent 


Respons 
e  Total 


Government  (.gov) 


Military  (.mil) 


Academia  (.edu) 

|  ■  •e'Mu-.rd' 

48% 

48 

Industry  (.com) 


28%  I  28 


%  'S-rU-’v'  Total  Respondents  I  100 


(skipped  this  question)  I  2 


Indicate  your  primary  business  sector: 


I'm  a  student 


University  of  Virginia 
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Eindhoven  Technical  Universii 


University  of  Virginia 

Department  of  Mathematical  Sciences  UNLV 
Micro  Analysis  &  Design 


universidad  del  sinu 

SBC  Communications  /  Telecom 

Hewlett  Packard 


ESI 

ESI 

ESI 


University  of  California 


Affiliation: 


Masters  Student 


Reseacher 


Sr.  Instructional  Designer 


Research  Scientist 


President  &  Sr.  Human  Factors  Eng. 


Computer  Scientist 


Website  intern 


Systems  Engineer 


coordinator  HSI 


HCI  Designer 


Training  Consultant 


Research  Assistant 


Professor  of  Computer  Science 


senior  instructional  designer 


Student 


Human  Factors  Engineer,  Cognitive  Scientist 


Assistant  Professor 


PhD  Researcher 


tech  assistant  and  project  manager 


senior  developer 


Senior  Developer 


Research  Analyst 


Professor 


Human  Computer  Interface  Engineer 


Manager,  User  Interface  Design 


Industrial  Hygienist 


Professor 


Human  Factors  Consultant 


Graduate  student 


Assistant  Professor 


User  Experience  Engineer 


Research  Associate 


Asst.  Professor 


Space  Human  Factors  Engineering  Program  Coordinator 


Assistant  in  Research 


Canada 


Virginia 


Florida 


minnesota 


Canada 


Maryland 


Michigan 


Canada 


Virginia 


Ontario,  Canada 


California 


Nevada 


New  Jerse 


Florida 


Washington  DC 


German 


Virginia 


Washington 


Spain 


Oregon 


Hawaii 


Hawaii 


Michigan 


Michigan 


NY,  USA 


Indiana 


Oregon 


Virginia 


Virginia 


California 


Maryland 


California  &  New  York 


Maryland 


The  Netherlands 


Virginia 


Nevada 


Colorado 


Colombia 


Texas 


Colorado 


MA,  USA 


California 


Question  17 


17.  Experience: 


How  many  years,  if  any,  experience  do  you  have 
in  modeling  human  performance? 


How  many  years,  if  any,  experience  do  you  have 
in  evaluating  user  interface  designs? 


Response 

Percent 


Response  Total 


(skipped  this  question) 


Experience: 


about  1 0,  off  and  on  among  other  things 


a  couple  of  months 


6  years 


1  year 


13+ 


0 


1  year 


4  years 


Depends  on  what  you  mean  by  "modeling  human  performance"  have  never  done  it  to  the  level  that  the 
seminar  described 


1 ,  as  a  qrad  student 


0  -as  it  was  outined  in  this  seminar 


si,  5  afios 


almost  none 


Experience: 


^R 


8  years 


1 .5  years  of  very  informal  occasional  evaluation 


40 


41 


mvself  on  day  1 ,  with  one  other  on  days  2  and  3 


just  me 


2-3  dependinq  on  the  day  and  time 


myself 


attended  alone 


various,  but  6-10 


If  you  have  additional  feedback,  please  enter  it  here: 


Wonderful  seminar... I  look  forward  to  others  to  come! 


complete  it  before  you  close  it. 


I  was  a  graduate  student  at  U.  of  Michigan  between  1995  and  1997,  and  I  took  a  course  with  Professor 
Kieras.  I  found  his  in-person  presentation  skills  to  be  very  good,  and  I  remember  I  always  looked  forward  to 
his  class.  In  this  "webcast"  format,  however,  he  came  across  as  somewhat  dry,  perhaps  because  he  couldn't 
gauge  the  reactions  of  his  audience  (and  he  seemed  restrained  in  his  use  of  humorous  anecdotes,  etc.).  I 
wonder  if  it  wouldn't  help,  in  general,  if  these  seminars  were  held  in  front  of  a  small  audience,  even  though 
they  are  being  broadcast  to  the  world  at  the  same  time. 


Registering  was  difficult  because  it  required  belonging  to  a  chat  group,  which  is  forbidden  by  my  company's 
network  security.  In  the  future,  please  offer  alternate  means  of  registration  when  announcing  seminars 


5.  The  slides  were  a  mirror  of  the  presentation.  The  presentation  added  nothing  for  me.  I  felt  like  I  was  listening 
to  an  audio  book. 


Please  consider  international  users  and  the  timing.  It’s  better  than  archiving.  Also  have  a  test  the  day  before 
to  make  sure  of  all  software  and  hardware  compatibility.  I  missed  on  a  seminar  because  the  software  they 
were  using  did  not  work  through  a  proxy  server. 


thought  the  slides  were  just  phrases  from  the  audio  that  were  very  plainly  presented.  When  the  audience  has 
the  speaker  to  see  and  engage  with  that's  acceptable.  However,  when  all  the  audience  sees  is  the  slides, 
there  should  be  more  than  just  parts  of  the  speech  in  printed  form.  **  just  a  thought! 


Reviews  and  surveys  of  this  sort  of  material  are  generally  invaluable  for  practicioners. 


Thank  you! 


When  presenting  a  topic  using  slide  presentation,  it  is  not  necessary  to  read  all  the  information  on  the  slides. 
I  rather  expect  more  elaboration  than  what  I  can  see  in  front  of  me. 


This  is  my  evaluation  about  the  online  seminar,  but  I  would  like  to  examine  the  material  posted  when  I  can.  I 
think  it  will  be  useful  to  open  a  comments  period  ... 


12.  Thanks  Lisa  for  arranging  this.  You  might  want  to  look  at  a  shareware  program  called  Babylon.  We  use  it  for 
some  distance  learning  applications  since  you  have  a  shared  whiteboard,  presenters  can  bring  up  their 
slides,  doodle  (in  fact,  participants  can  doodle),  and  do  text  chat  all  at  the  same  time. 
http://visopsys.org/andy/babylon/ 


13.  I  tried  to  join  the  chat  session  but  was  unable  to  do  so  because  the  maximum  number  of  participants  had 
been  met.  It  would  have  been  nice  to  have  been  able  to  join  and/or  download  a  chat  record. 


Excellent  course  - 


It  was  a  great  forum  for  learning!  Wish  I  was  able  to  attend  them  all! 


I  thought  that  David's  example  was  very  difficult  to  follow.  It  was  hard  to  read  without  magnifying  it.  Switching 
back  and  forth  to  different  areas  that  were  not  immediately  obvious  or  within  the  picture,  was  difficult.  Overall, 
the  seminar  was  terrific.  Thanks  to  the  presenters  for  being  so  well  prepared  and  knowledgeable. 


I  thought  it  worked  much  better  to  have  one  instructor  answering  questions  in  the  chat  while  they  were  still  in 
context  than  to  have  all  questions  held  until  the  next  break.  I  think  it's  possible  a  different  forum  than  yahoo 
would  work  better,  like  an  ICQ  chat  room  or  something  -  something  that  supports  the  copying  and  pasting  of 
text  off  the  chat  history,  and  logs  it,  and  lets  an  unlimited  number  of  people  connect.  I  was  quite  pleased  that 
someone  from  my  alma  mater  (uwaterloo)  was  first  to  offer  to  mirror  the  files  when  they  couldn't  be 
downloaded  from  yahoo.  I  hope  all  these  technical  lessons  have  been  noted  for  the  future.  Thanks  for 
organizing  it! 


Good  start  of  a  great  form  of  interaction... 


I  found  it  inconvenient  using  Yahoo  groups  to  register  for  the  conference  and  to  download  conference 
materials. 


I  was  impressed  with  the  speakers'  knowledge  of  content.  I  hope  they  continue  this  service.  Having  a 
seminar  on  line  is  great,  it  saves,  time  and  ....  to  participate.  I  hope  you  continue  this  in  the  future.  Thanks 


Overall,  great  job!,  I  really  appreciate  the  effort  you  all  put  into  this  seminar.  It  worked  out  great  and  was  very 
informative.  Nonetheless  in  hopes  of  continued  improvements.  It  would  be  of  great  value  if  future  seminars 


*  M  -C 


had  a  live  visual  component.  Even  though  the  audio  is  just  fine,  1  felt  1  needed  a  dynamic  mechanisms  of 
letting  me  know  where  in  the  discussions  we  were  (ex.  which  slide  we  are  in,  and  what  point  in  the  slide  we 
are  talking  about).  Also,  when  doing  demos  of  software  applications  it  is  very  hard  to  understand  where  in  the 
screen  are  the  items  the  speaker  is  talking  about,  without  some  kind  of  'pointing  device".  Note  also  that  live 
demos  of  software  applications  done  through  live  online-video  are  very  lousy  too  (the  pointers  and  screen 
labels  are  impossible  to  see,  due  to  the  resolution.  1  suggest  that  a  combination  of  the  slides  an  video  could 
enhance  the  sessions.  UCF  has  the  largest  long  distance  learning  program  in  Florida,  1  am  almost  certain 
they  would  let  you  use  their  facilities  for  live-video  streaming.  Also,  1  am  had  difficulty  joining  the  online-chat 
due  to  number  of  participants  limitations. 

1  22.  I  Thanks  for  teaching  me  somethinq  new.  1  will  be  able  to  use  these  methods  to  think  about  other  issues.  I 

This  was  a  very  well  run  seminar  and  1  felt  well  prepared.  The  topic  was  very  informational  and  the  speakers  1 
were  very  knowledgeable.  Thank  you  for  the  opportunity.  | 

r 

allow  more  people  in  chat  or  open  more  than  one  chat  forum  Special  THANK  YOU  to  the  Office  of  Naval 
Research  and  to  everyone  involved  in  presenting  this  seminar. 

Thanks  for  the  seminar.  Thanks  for  your  dedication  to  solve  every  problem.  Congratulations.  It  was 
worthwhile 

I  26.  i  1  appreciated  the  fact  that  the  seminar  fell  within  normal  business  hours  even  in  Hawaii  Standard  Time,  1 

27, 

This  is  my  first  time  taking  a  seminar  like  this  and  my  first  "formal"  time  of  any  teaching  on  cognitive 
modeling/  architecture.  1  am  very  interested  and  plan  to  be  studying  more  on  cognitive  psychology,  human 
factors,  computer  programming/graphics,  aviation  and  military  simulation  and  training  as  a  student  at  Purdue. 
Thank  you  for  this  great  opportunity  and  1  would  love  to  be  included  in  any  future  opportunities  along  these 
lines.  As  1  progress  in  my  skills  and  studies,  1  hope  to  be  able  to  meet  some  of  these  folks,  attend  seminars 
and  make  applications  to  our  research  endeavor.  1  have  some  background  in  aviation  education,  a  private 
pilot  license  and  was  an  air  traffic  controller  in  military.  Now  studying  computer  graphics  and  interested  in 
training  and  simulation.  If  you  had  any  input  on  what  types  of  seminars  and  training  that  would  relate  to  this, 
could  you  please  contact  me?  Thank  you  so  much! 

F 

Thanks  for  opening  this  up  to  the  public,  1  feel  1  gained  a  good  deal  of  knowledge  and  exposure  via  this  1 

opportunity.  | 

f 

1  really  like  the  prompt  posting  of  the  audio  archives.  1  also  found  the  posting  of  questions  answered  off-line  I 
useful.  1  would  like  to  see  an  archive  of  the  text  chat.  | 

30. 

Attempts  to  ask  question  via  phone  failed,  not  sure  why.  Glad  to  see  multiple  paths  provided  for  questions. 

Also  valued  multiple  alternatives  fordownloading  materials.  The  use  of  a  facility  like  Elluminate  would  have 
made  attendance  easier  (vs  downloading  and  tracking  multiple  files).  Overall,  was  very  impressed  with  the 
instructors  and  their  blend  of  theoretical/practical  experience  and  breadth  and  depth  of  knowledge  of  the 
subject  and  current  research.  The  course  organization  was  also  very  professional,  not  least  the  moderator’s 
grace  and  flexibility  in  dealing  with  emergent  challenges. 

H 

It  would  be  great  if  the  slides  were  in  PowerPoint  format  for  printing/following  along.  It  would  also  be  helpful  if 
we  could  have  had  the  slides  presented  online,  synchronized  with  the  talk 

n 

System  requirements  for  the  seminar  should  be  stated  up  front.  The  streaming  audio  didn't  work  on  a  Mac 
(Vivavid  said  they  support  Macs  but  the  conference  wasn't  set  up  for  them).  1  think  the  chat  required  Java 

Virtual  Machine  software. 

0 

overall,  excellent  experience!  I'm  very  pleased  that  this  was  a  free  seminar — 1  didn't  have  to  convince  my 
boss  to  pay  for  anything! 

|  34.  I  Nice  Seminar,  1  am  glad  that  1  knew  about  it.  1 

1  35.  1  Playing  back  with  Video  would  be  better  for  me.  1 

36. 

HU  1 

Great  opportunity,  definitely  would  like  to  see  this  on  a  semi-regular  basis.  Would  prefer  slightly  more  lively 
presentation  than  Dave  (1  could  have  read  his  slides  with  more  verve  than  he  read  his  slides).  Definitely  liked 
the  interaction  opportunities  with  questions  via  phone  and  chat. 

H 

Seminar  description  should  indicate  that  most  of  the  work  being  presented  is  still  in  the  research  stage.  It's 
clear  that  a  simulated  user  approach  is  useful,  especially  for  very  complex  software  applications.  1  had  hoped 

44 
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to  learn  something  adaptable  to  less  complex  but  more  ubiquitous  interfaces  where  error  rates  may  not  be 

Summary  of  all  the  tools  was  nice.  Maybe  include  slides  with  examples  of  projects  that  have  used  each  of 
the  tools/cognitive  architectures. 

I'm  very  pleased  with  the  ambitions  of  the  organisation  that  provided  the  seminar.  Making  this  a  world-wide 
Thanks! 

It  isn't  clear  who  the  intended  audience  is.  Sometimes  I  think  it’s  a  basic  HFE  class  and  other  times  it  seems 
like  it  is  for  those  operating  in  the  field. 

I  really  liked  this  format.  It  is  very  convenient  and  the  timing  worked  very  well  for  me.  l  am  really  getting  a 
very  good  idea  about  this  topic.  I  hope  that  there  is  some  software  available  to  try  it  out.  I  haven't  heard 
Bonnie's  segment  yet.  I  thought  this  survey  was  for  the  first  day.  There  is  no  way  to  'undo'  Bonnie's  ratings. 
(interesting  usability  issue!) 

Question  20 

Optional:  If  you  would  like  to  be  on  a  mailing  list  for  further  seminars,  please  enter  your  name _ _ 


Sebastien 


Stephen  Mallett 


kirschenbaumss@npt.nuwc.navy.mil 


hsieh@tamu.edu 


melissa@accessfulsolutions.com 


brock@itd.nrl.navy.mil 


afineline@pathcom.com 


sm4282@drexel.edu 


ahoo.com 


iohn.e.ross@navy.mil 


nmadrid@ugr.es 


brent@CSUFresno.edu 


macdonaldJo@idsi.com 


joseph.divita@navy.mil 


han@osweqo.edu 


clairemcandrewl  @yahoo.com 


raquilina@mail.arc.nasa.qov 


mike  mailings@mac.com 


dho001@umaryland.edu 


rozanski@it.rit.edu 


iH 


soartech.com 


ergo@vt.edu 


tweekes@fit.edu 


mit.edu 


jbos@humansys.com 


akessell@stanford.edu 


drfsaba2000@yahoo.com 


mgalvarez999@yahoo.com 


rchampne@mail.ucf.edu 


tillmans@iit.edu 


bonnie.h.battaqlia@nasa.qov 


doconnor@lpg.fsu.edu 


abartel@uos.de 


bmorinaka@comcast.net 


jon@usability-archtiects.com 


valbuen@ugr.es 


48 


uantumleap.us 


cmanly@connectimaging.com 


PHK 


ahoo.com 


lhamel@soartech.com 


shenty@alumni.ucsd.edu 


igel@purdue.edu 


mfelc@allstate.com 


aranda  r@arandaconsulting.com 


danhorn@hqda.army.mil 


mmatessa@arc.nasa.gov 


Philip  Hove@whirlpool.com 


aahmad@mail.ucf.edu 


bsoh@vt.edu 


wanqhui@ios.cn 


kevin.cropper@jhuapl.edu 


olitemachines.com 


johnsrude@cs.orst.edu 


mail.com 


uerlain@virqinia.edu 


drfsaba2000@yahoo.com 


bhautamaki@maad.com 


edgaralarcon@unisinu.edu.co 


knott@labs.sbc.com 


rozanski@it.rit.edu 


'khoosh@cal.berkeley.edu 
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